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Abstract 

In this paper, we study the upper and the lower bounds on the joint source-channel coding error 
exponent with decoder side-information. The results in the paper are non-trivial extensions of the Csiszar's 
classical paper [5]. Unlike the joint source-channel coding result in [5], it is not obvious whether the 
lower bound and the upper bound are equivalent even if the channel coding error exponent is known. 
For a class of channels, including the symmetric channels, we apply a game-theoretic result to establish 
the existence of a saddle point and hence prove that the lower and upper bounds are the same if the 
channel coding error exponent is known. More interestingly, we show that encoder side-information does 
not increase the error exponents in this case. 

I. Introduction 

In Shannon's very first paper on information theory [11], it is established that separate coding is optimal 
for memoryless source channel pairs. Reliable communication is possible if and only if the entropy of 
the source is lower than the capacity of the channel. However, the story is different when error exponent 
is considered. It is shown that joint source-channel coding achieves strictly better error exponent than 
separateQ coding [5]. The key technical component of [5] is a channel coding scheme to protect different 
message sets with different channel coding eiTor exponents. In this paper, we are concerned with the 
joint source-channel coding with side information problem as shown in Figure [T] For a special setup of 
Figure [T] where the discrete memoryless channel (DMC) is a noiseless channel with capacit}jl R, i.e. the 
source coding with side-information problem, the reliable reconstruction of a" at the decoder is possible 
if and only if R is larger than the conditional entropy H{Pj^\b) [13]. The error exponents of this problem 
is also studied in [8], [6] and more importantly in [1]. 
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Fig. 1. Source coding with decoder side-information 



The duality between source coding with decoder side-information and channel coding is established in 
the 80's [1]. This is an important result that all the channel coding error exponent bounds can be easily 
applied to source coding with side-information error exponent. The result is a consequence of the type 
covering lemma [6], also known as the Johnson-Stein-Lovasz theorem [4]. With this duality result, we 
know that the error exponent of channel coding of channel Wy\x with channel code composition Qx is 
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*In [5], Csiszar hand-wavily shows that the obvious separate coding scheme is suboptimal in terms achieving the best error 
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essentially the same problem as the error exponent of source coding with decoder side-information where 
the joint distribution is Qx x W^y|x- Hence a natural question is what if we put these two dual problems 
together, what is the error exponent of joint source-channel coding with decoder side-information? 

The more general case, where Wy\x is a noisy channel, is recently studied [15], [14]. It is shown 
that, not surprisingly, the reliable reconstruction of a" is possible if and only if the channel capacity 
of the channel is larger than the conditional entropy of the source. A suboptimal error exponent based 
on a mixture scheme of separate coding and the joint source channel coding first developed in [5] is 
achieved. In this paper, we follow Csiszar's idea in [5] and develop a new coding scheme for joint source 
channel coding with decoder side-information. For a class of channels, including the symmetric channels, 
the resulted lower and upper bounds have the same property as the joint source-channel coding error 
exponent without side-information in [5]: they match if the channel coding error exponent is known at a 
critical rate. We use a game theoretic approach to interpret this result. 

The outline of the paper is as follows. We review the problem setup and classical error exponent 
results in Section |lll Then in Section JIIJ we present the error exponent result for joint source-channel 
coding with both decoder and encoder side information which provides a simple upper bound to the error 
exponent investigated in the paper. This is a simple corollary of Theorem 5 in [5]. The main result of 
this paper is presented in Section JV] Some implications of these bounds are given in Section |Vl 

II. Review of source and channel doing error exponents 

In this paper random variables are denoted by a and b, the realizations of the random variables are 
denoted by a and h. 

A. System model of joint source-channel coding with decoder side-information 

As shown in Figure[T] the source and side-information, and respectively, are random variables i.i.d 
from distribution Pab on a finite alphabet AxB. The channel is memoryless with input/output probability 
transition Wy\X' where the input/output alphabets X and y are finite. Without loss of generality, we 
assume that the number of source symbols and the number of channel uses are equal, i.e. the encoder 
observes a'^ and sends a codeword x^{a'^) of length n to the channel, the decoder observes the channel 
output and side-information 6" which is not available to the encoder, the estimate is a"(6",y"). 

The error probability is the expectation of the decoding error average over all channel and source 
behaviors. 

Pr(a" a"(b",y")) = J] PabIo", 6") H^y|x(2/"|x"(a"))l(a" a"(6", y")). (1) 

a" ,6" y" 

The error exponent, for the optimal coding scheme, is defined as 

EiPAB,WYix) = '^^ inf -ilogPr(a"/a"(6",y")). (2) 

The main result of this paper is to establish both upper and lower bounds on E{Pab, Wy^x) and show 
the tightness of these bounds. 

B. Classical error exponent results 

H We review some classical results on channel coding error exponents and source coding with side- 
information error exponents. These bounds are investigated in [9], [6], [8] and [7]. 

^ In this paper, we write the error exponents (both channel coding and source coding) in the style of Csiszar's method of 
types, equivalent Gallager style error exponents can be derived through the Fenchel duality. 
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1) Channel coding error exponents Ec{R,Wy\x)' Channel coding is a special case of joint source- 
channel coding with side-information: the source a and the side-information b are independent, i.e. 
Pab = Pa X Pb, and a is a uniform distributed random variable on {1, 2, 2^}. For the sake of 
simplicity, we assume that 2^ is an integer. This is not a problem if 2^ is not an integer since we can 
lump K symbols together and approximate 2^^ by an integer for some K, this is not a problem because 

lim log2 ( [2^^J ) = R. With this interpretation of channel coding, the definitions of error probability 
in ([B and error exponent in Q still holds. 

The channel coding error exponent Ec{R,Wy\x) is lower bounded by the random coding error 
exponent and upper bounded by the sphere packing error exponent. 

Er{R, Wy\x) < Ec{R, Wy\x) < Esp{R, H^y|x) (3) 
where Er{R,WY\x) = max inf D{Vy\x\\Wy\x\Sx) + \I{Vy\x;Sx) - (4) 

Sx Vy\x 

= m.&yiEr{R,Sx,WY\x) 

Sx 

Esp{R,Wy\x) = max inf D{Vy\x\\Wy\x\Sx) (5) 

= max£'sp(-R, 5x, WV|x) 

Sx 

Here Sx is the input composition (type) of the code words. Er{R, Wy\x) = Esp{R, Wy\x) in the high 
rate regime that R > Rcr where Rcr is defined in [9] as the minimum rate for which the sphere packing 
Esp{R,Wy\x) and random coding error exponents E.r{R,WY\x) match for channel Wy\x- There are 
tighter bounds on the channel coding error exponents Ec{R, Wy\x) in the low rate regime for R < Rcr, 
known as straight-line lower bounds and expurgation upper bounds [9]. However, in this paper, we focus 
on the basic random coding and sphere packing bounds, as the main message can be effectively carried 
out. 

It is well known [9] that both the random coding and the sphere-packing bounds are decreasing with 
R and are convex in R. And they are both positive if and only if i? < C{WY\x)y where C{Wy\x) is 
the capacity of the channel Wy\x- 

2) Source coding with decoder side-information error exponents: This is also a special case of the 
general setup in Figure [T] This time the channel Wy\x is a noiseless channel with input-output alphabet 
X = y and \X\ = 2^. Again, we can reasonably assume that 2^ is an integer. 

The source coding with side-information error exponent e{R,PAB) can be bounded as follows: 

e^R, Pab) < e{R, Pab) < eu{R, Pab) (6) 



where eUR^PAB) = inf D{Qab\\Pab) + \R - H{Qaib)\^ 

Qab 

eu{R,PAB)= inf D{Qab\\Pab)- 

Qab'-H(Qa\b)>R 

The duality between channel coding and source coding with decoder side information had been well 
understood [1]. We give the following duality results on error exponents. . 

eiR,QA,PB\A) = Ec{H{Qa)-R,Qa,Pb\a) 
or equivalently : e{H{QA) - R, Qa, Pb\a) = Ec{R, Qa, Pb\a) 

"* In this paper, if i? > logj |.4| for source coding with side-information error exponents, we let the error exponent be oo. 
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where Ec{R, Qa, Pb\a) is the channel coding error exponent for channel Pb\a r^t^ ^ ^"d the codebook 
composition is Qa- Qaj -Pb|a) is the source coding with side information error exponent at rate R 
with source sequences uniformly distributed in type Qa and the side information is the output of channel 
Pb\a with input sequence of type Qa- So obviously, we have: 

Ec{R, Pb\a) = max{E,iR, Qa, Pb\a)} 

Qa 

e{R,PAB) = min{D{QA\\PA) + eiR,QA,PB\A)} 

Qa 

These results are established by the type covering lemma [5] on the operational level, i.e. a complete 
characterizations of the source coding with side information error exponent e{R,QA,PB\A) impUes a 
complete characterizations of the channel coding error exponent Ec{H{Qa) — R,Qa,Pb\a) and vice 
versa. 

From these duality results, it is well known that both the lower and the upper bounds are increasing 
with R and are convex in R. And they are both positive if and only if i? > H{Pa\b)- The special case 
of the source coding with decoder side information problem is that the side information is independent 
of the source, i.e. Pab = Pa x Pb- In this case, the error exponent is completely characterized [6], 

e{R,PA)= inf D{Qa\\Pa) (7) 

Qa:H{Qa)>R 

3) Joint source-channel coding error exponents [5]: In his seminal paper [5], the joint source-channel 
coding error exponents is studied. This is yet another special case of the general setup in Figure [T] When 
a and b are independent, i.e. Pab = Pa x Pb, we can drop all the b terms in ([1]). Hence the error 
probability is defined as: 

Pr(a" / a"(y")) = J] P^(a") ^ VFy|x(y"|:r"(a"))l(a" / a"(y")). (8) 

a" y" 

Write the error exponent of ([8]) as E{Pa,Wy\x)- The lower and upper bounds of the error exponents 
are derived in [5]. It is shown that: 

min{e(i?, Pa) + Esp{R, Wy\x)} < E{Pa, Wy\x) < mm{e{R, Pa) + Er{R, Wy\x)} (9) 

The upper bound is derived by using standard method of types argument. The lower bound is a direct 
consequence of the channel coding Theorem 5 in [5]. 

The difference between the lower and upper bounds is in the channel coding error exponent. The joint 
source channel coding error exponent is "almost" completely characterized because the only possible 
improvement is to determine the channel coding error exponent which is still not completely characterized 
in the low rate regime where R < R^r- However, let R* be the rate that minimizes {e(i?, Pa) + 
Er{R, Wy\x)}, if R* > Rcr or equivalently Er{R*, Wy\x) = EspiR* ,Wy\x), then we have a complete 
characterization of the joint source channel coding error exponent: 

E{Pa, Wy\x) = e{R*,PA) + Er{R*, Wy\x)- (10) 

The goal of this paper is to derive a similar result for E{Pab, Wyix) defined in ([2]) as that for the joint 
source channel coding in ^ and ( fTOb . 
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4) A recite of Theorem 5 in [5]: Given a sequence of positive integers {m„} with ^logm„ — > 
and m„ message sets Ai, ....Am^ each with size \Ai\ = 2"^'. Then there exists a channel code (/o, (t>o), 
where the encoder /o : Ui!Lo-^i ~^ where /o(a) = a;"(a) € for a G Ai and the decoder 
4>o'-y^ ^ UilLo-^j' write 4>o{y"') as a(y") s.t. for any message a G Ai, the decoding error 

Pe{a) = ^l^y|x(y"|x"(a))l(a / a(y")) < 2-(^'^(«"^-'^-i-)-^") 

for every channel Wy\x^ and e„ ^ 0. In particular, if the channel Wy\x is known to the encoder, each 
S^^ can be picked to maximize Er{Ri, S^^, WY\x)y hence for each a ^ Ai'. 

Pe{a) < 2"(^-(«-^-i-)-^"). 

This channel coding theorem as Csiszar put it, the "main result of this paper" in [5]. We use this theorem 

directly in the proof of the lower bound in Proposition [T] and further modify it to show the lower bound 
in Theorem [U 

III. Joint source-channel coding error exponent with both decoder and encoder 

SIDE-INFORMATION 

As a warmup to the more interesting scenario where the side-information is not known to the encoder, 
we present the upper/lower bounds when both the encoder and the decoder know the side-information. 
This setup is shown in Figure |2l 
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DMC Wyix 




Decoder 















{ai, bi) ~ Pab 
I 

6" ' ' 

Fig. 2. Source coding with both decoder and encoder side-information 
The error probability of the coding system is, similar to ([T]): 

The error exponent of this setup is denoted by Eboth{PAB, Wy^x) which is defined in the same way as 
E{Pab, Wyix) in @- The difference is that the encoder observes both source a" and the side-information 
6", hence the output of the encoder is a function of both: x"(a",6"). So obviously, EhothiPAB^Wy^x) 
is not smaller than E{Pab,Wy\x)- 

Comparing (ITTI) and ([8]l, we can see the connections between joint source-channel coding with both 
decoder and encoder side information and joint source-channel coding. Knowing the side information 6", 
the joint source channel coding with both encoder and decoder side information problem is essentially 
a channel coding problem with messages distributed on A"" with a distribution P4|^(a"|6"). Hence we 
can extend the results for joint source-channel coding error exponent [5]. We summarize the bounds on 
EbothiPAB^Wyix) in the following proposition. 
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Proposition 1: Lower and upper bound on EbothiPAB, Wy\x) 

E 

EbothiPAB, Wyiz) > mm{euiR,PAB) + EriR,WYix)} (12) 



EbothiPAB, Wyix) < min{ec/(i?, Pab) + E^piR, Wyix)} 

R 



R 

Not explicitly stated, but it should be clear that the range of R is (0,log2 

Proof: see Appendix lAl Because EbothiPAB, Wy\x) is no smaller than EiPAB, Wy\x), so the lower 
bound of EiPAB, Wy\x) in Theorem [T] is also a lower bound for EbothiPAB, Wy\x)- However, in the 
appendix, we give a simple proof of the lower bound on EbothiPAB, Wy\x) which is a corollary of 
Theorem 5 in [5]. □ 

Comparing the lower and the upper bounds for the case with both encoder and decoder side-information, 
we can easily see that if R* minimizes {cuiR, PAB)~^EriR, W^y\x)} smd EgpiR* , ^Vy\x) — E^iR* , W^y|^), 
then the upper bound and the lower bound match. Hence, 

EbothiPAB, Wyix) = euiR*, Pab) + EriR*, Wyix)- (13) 
In this case EbothiPAB, Wy\x) is completely characterized. 
IV. JOINT SOURCE-CHANNEL ERROR EXPONENTS WITH ONLY DECODER SIDE INFORMATION 

We Study the more interesting problem where only decoder knows the side-information in this section. 
We first give a lower and an upper bound on the error exponent of joint source-channel coding with 
decoder only side-information. The result is summarized in the following Theorem. 

Theorem 1: Lower and upper bound on the joint source channel coding with decoder side-information 
only, as setup in Figure [TJ error exponent: For the error probability Pr(a" ^ a"(£)"^,y")) and error 
exponent EiPAB, Wy\x) defined in ([T]) and we have the following lower and upper bounds: 

EiPAB, Wy\x)> (14) 
min max miri {DiQAB\\PAB) + DiVY\x\\WY\x\SxiQA)) + \IiSxiQA);VY\x) " ^(^^1^)1+} 

Qa Sx(Qa)Qb\a,Vy\x 

EiPAB, Wy\x)< (15) 
min max min {DiQAB\\PAB) + DiVY\x\\WY\x\SxiQA))} 

Qa Sx{Qa)Qb\a,Vy\x.I{Sx{Qa);Vy\x)<H{Qa\b) ' ' 

Proof: The main technical tool used here is the method of types. For the lower bound we propose a 
joint coding scheme for the joint source channel coding with side information problem. This scheme is 
a modification of the coding scheme first proposed in [5]. However, we cannot directly use the channel 
coding Theorem 5 in [5] because of the presence of the side information. In essence, we have to study 
a more complicated case using the method of types. Details see Appendix |B] □ 
To simplify the expressions of the lower and upper bounds and later give a sufficient condition for 
these two bounds to match, we introduce the "digital interface" R and have the following corollary. 

Corollary 1: upper and lower bounds on EiPAB, Wy\x) with "digital interface" R 

EiPAB, Wy\x) < min max mm{euiR, Pab, Qa) + EspiR, SxiQA), Wy\x)} (16) 

Qa Sx(Qa) R 

EiPAB,WY\z)>^^^ max vainieuiR, Pab, Q a) + EriR, SxiQA), Wy\x)} (17) 

Qa Sx(Qa) R 

where EriR, SxiQA), Wy\x) is the standard random coding error exponent for channel Wy\x rate R 
with input distribution SxiQA) defined in dD, while euiR, Pab, Qa) is a peculiar source coding with 
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side-information eiTor exponent for source Pab at rate R, where the empirical source distribution is fixed 
at Qa- That is for Qa 

eu{R,PAB,QA) = min D{Qab\\Pab) (18) 

Qb\a'-H{Qa\b)^R 

Proof: The proof is in Appendix O □ 

With the simplified expression of the lower and upper bounds in Corollary [T] we can give a game 
theoretic interpretation of the bounds. And more importantly, we present some sufficient conditions for 
the two bounds to match. 

A. A game theoretic interpretation of the bounds 

The lower and upper bounds established in Corollary [1] clearly have a game theoretic interpretation. This 
is a two player zero sum game. The first player is "nature", the second player is the coding system, the 
payoff from "nature" to the coding system is the bounds on the error exponents in Corollary [T] "Nature" 
chooses the marginal of the source Qa (observable to the coding system) and R which is essentially 
the side information Qb\a and the channel behavior Vy\x (non-observable to the coding system). The 
coding system choose Sx{Qa) after observing Qa- Hence in this game, the "nature" has two moves, 
the first move on Qa and the last move on R which is essentially Qb\a and Vy\x^ while the coding 
system has the middle move on Sx{Qa)- 

Comparing Corollary [T] for joint source-channel coding with decoder side information and the classical 
joint source-channel coding error exponent [5] in Q, it is desirable to have a sufficient condition that the 
lower bound and the upper bound match, i.e. the complete characterization as that in (ITOl ). It is simpler 
for the case in ^ since all is needed is that the sphere backing bound and the random coding bound to 
match at the critical rate R* as discussed in Section III-B.3I However, for the two bounds in Corollary [T] 
it is not clear what the conditions are such that these two bounds match. Suppose that the solution of 
the game (1161 ) is [Q^, S^{Q a) , R^) and solution of the game (ini) is {Q\,S^^{Qa),R^). An obvious 
sufficient condition for the two bounds match is as follows: 

{Q'a.S'x{Qa).R') = m,SJciQA),Rn and P^y|x) = S^(Qa), H^y|x) (19) 

This condition is hard to verify for any source channel pairs. In the next section, we try to simplify the 
condition under which these two bounds match for a class of channels. 

B. A sufficient condition to reduce min{max{min{-}}} to min{-} 

The difficulty in studying the bounds in Corollary [T] is that the min and max operators are nested. 
The problem will be simplified if we can change the order of the min and max operators. 

Corollary 2: For symmetric channels Wy\x defined on Page 94 in [9], this includes the binary 
symmetric and binary erasure channels, where the input distribution Sx to maximize the random coding 
error exponent Er{R, Sx ,Wy^x) is uniform on X, or for more general channel^ where the input 
distribution Sx to maximize the random coding error exponent Er{R, Sx, Wy\x) is the same for all R, 
then the upper and lower bounds in Theorem [T] and Corollary [T] can be further simplified to the following 
forms: 

EiPAB, Wyix) < mm{euiR, Pab) + Esp{R, Wy\x)] (20) 
E{Pab,Wy\z) > mm{eu{R,PAB) + Er{R,WY\x)} (21) 

R 

^For example, a channel consisted of parallel symmetric channels. 
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Note: in this case, the upper and lower bounds for E{Pab,Wy\x) is the same as those for E^othiPAB ■,Wy\x) 
in Proposition |7] More discussions see Section [3 

Proof: An important property for symmetric channels is that the input distribution that maximizes 
the random coding error exponent is constant for all rate R, hence the inner maxmin{ } is equal to 
minmaxj-}, i.e. 



E[Pab,Wy\x) > mill max min{et/(i?, P^B, Qa) + ^r(i?, Sx(Qa), VFy|x)} 

Qa Sx{Qa) R 

= minmin max {euiR, Pab,Qa) + EriR, Sx{Qa),Wy\x)} 

Qa R SxiQA) 

= mmmm{eu{R,PAB,QA) + Er{R,WY\x)} (22) 

Qa R 

= min{min{ec/(ii, Pab,Qa)} + Er^R, Wy\x)} 

R Qa 

= mm{eu{R,PAB) + Er{R,WY\x)} (23) 
-ft 

where (l22l ) follows the definition of random coding bound in (O and (l23l) follows the obvious equality: 

mmeu{R,PAB,QA) = min D{Qab\\Pab) = eu{R, Pab)- 

Qa Qab.H{Qa\b)>R 

The upper bound in [20] is trivial by noticing that maxminj-} < minmax{-} [2], hence: 

E{Pab,Wy\x) < rmn max mm{euiR, Pab, Q a) + Esp{R, SxiQA), Wyix)} 

Qa Sx{Qa) R 

< minmin max {eu{R, Pab, Qa) + EspiR, SxiQA), Wy\x)} 

Qa R Sx{Qa) 

= mmminieuiR, Pab, Q a) + EspiR,WY\x)} 

Qa R 

= min{min{et/(i?, Pab, Qa)} + EspiR, Wy\x)} 

R Qa 

= mm{eu{R,PAB) + Esp{R,WYix)} (24) 
-ft 

Corollary |2] is proved. □ 

With this corollary proved, we can give a sufficient condition under which the lower bound and upper 
bound match similar to that for the joint source-channel coding case in Section lTl-B.3l More discussions 
see Section IVl 

C. Why it is hard to generalize Corollary \2\ to non-symmetric channels? 
Whether max min{et/(i?, -Pab, (5yi)+-E'r(-R, 'S'x(Qyi), WV|x)} is equal to min max {eu{R, Pab, Q a) + 

Sx(Qa) R R Sx{Qa) 

EriR, Sx{Qa),Wy\x)} is not obvious for general (non-symmetric) channels. A sufficient condition of 
the existence of a unique saddle point hence the equality is known as the Sion's Theorem [12] which 
states that: 

max min f(n, u) = mm max f(u, u) (25) 

if M. and N are convex, compact spaces and / a quasi-concave-convex (definitions see [2]) and continuous 
function on x M . For the function of interest,: 

max mm{euiR,PAB,QA)^EriR,SxiQA),WY\x))- (26) 

Sx{Qa) R 
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We examine the sufficient condition under which a unique equilibrium exists, according to the Sion's The- 
orem. First, eu{R, Pab, Qa)+Esp{R, Sx{Qa), ^y|x) is quasi-convex in R because both eu{R, Pab, Qa) 
and Esp{R, Sx{Qa),Wy\x) convex, hence quasi-convex in R. However, (l26l) is not quasi concave 
on SxiQA)- 

Er{R,Sx{QA),WYix) = inf DiVY\x\\WYix\Sx{QA)) + \I{Vy\x;Sx{Qa)) - 

Vy I A- 

notice that the first term is linear in SxiQA), the second term is quasi-concave but not concave. But 
the sum of a linear function and a quasi-concave function might not be quasi-concave. This shows that 
the minmax theorem cannot be established by using the Sion's Theorem. This does not mean that the 
min max theorem cannot be proved. However for a non quasi-concave function that may have multiple 
peaks, minmax{-} is not necessarily equal to maxmin{-}. 

V. "Almost" complete characterization of E{Pab,Wy\x) for symmetric channels 
The sufficient condition in Corollary [2] is important, since binary symmetric and binary erasure channels 
are among the most well studied discrete memoryless channels. We further discuss the implications of 
the "almost" complete characterization of E{Pab,Wy\x) for symmetric channels. 

First we give an example shown in Figure [3] and Figure IH The source a is a Bernoulh 0.5 random 
variable and the joint distribution has the distribution 

r 0.50 0.00 \ 
= I 0.05 0.45 / ^^^^ 

The channel Wy\x is a binary symmetric channel with cross rate 0.025. The channel coding error 
exponent bounds Er{R, Wy\x) and Esp{R, Wy\x) and the source coding with decoder side-information 
upper bound eu{R, Pab) are plotted in Figure [3] The channel coding bound match while R > R^r, 
where Rcr is defined in [9]. 

Note: the lower bound of the source coding with side information error exponent eL{R, Pab) i^ not 
plotted in the figure. 

In Figure IH we add both the lower and upper bounds on the joint source channel coding with decoder 
side information to the plot in Figure[3] For this source channel pair Pab and Wy|x, we have a complete 
characterization of Ei)oth{PAB,WY\x) because the channel is symmetric and the two bounds match at 
the minimal point, i.e. the two curves: eu{R,PAB) + Esp{R,Wy\x) and eu{R,PAB) + -E'r(-R, 
match at the minimal point as shown in Figure |4l The value of the minimum is Ej shown in Figure ID 

A. Encoder side information often does not help 

Similar to Proposition [TJ we can see the conditions under which we can give a complete characterization 
of the joint source channel coding with decoder only side information error exponent E{Pab,Wy\x)- 
If R* minimizes {eu{R,PAB) + Er{R,WY\x)} and Esp{R*,Wy\x) = Er{R*,WY\x), then the upper 
bound and the lower bound match. Hence: 

E{PAB,WY\x) = eu{R*,PAB) + EriR*,WY\x)- (28) 

Comparing Corollary |2] and Proposition [T] we bound the error exponent with or without decoding 
side-information by the same lower and upper bounds. This does not mean that E{Pab,Wy\z) = 
Eboth{PAB,WY\z) always holds. But if the lower bound and upper bound match, which is shown in 
Figure |4l then we have: 

E{Pab,Wyiz) = EbothiPAB, Wy\z) = eu{R\PAB) + Er{R\ Wyix)- (29) 
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Fig. 3. The upper bound on source coding withi side- 
information error exponent eu{R,PAB) is the dotted 
line. The random coding bound Er{R, Wy\x) and sphere 
packing bound Esp{R,Wy\x) for channel coding error 
exponents are the solid line and the dashed line respec- 
tively. 



Fig. 4. euiR, Pab)+Esp(R, Wy\x) and eu{R, Pab) + 
Er{R,WYix) are added to Figure [3] in dashed line and 
solid line respectively, they match at the minimal point 
hence the joint source-channel coding with decoder side- 
information error exponent is completely determined as 
E{Pab,Wy\x) = Ej And Es is the separate coding 
error exponent Eseparate{PAB,WY\x) defined in {33). 



where R* minimizes eu{R,PAB) + Er{R,WY\x) ^nd R* > i?cr- This is another example for block 
coding where knowing side-information does not help increase the error exponent. In the contrary, as 
discussed in [3], in the delay constrained setup, there is a penalty for not knowing the side-information 
even if the channel is noiseless. 

B. Separate coding is strictly sub-optimal 

An obvious coding scheme for the problem in Figure [T] is to implement a separate coding scheme. 
A source encoder first encodes the source sequence a" into a rate R, where R is determined later, bit 
stream c"'^{aP) then an independent channel encoder encodes the bits c"^ into channel inputs x". The 
channel decoder first decodes the channel output into bits c"^ and then the independent source decoder 
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reconstructs a" from c" and side information 6". This is a separate coding scheme with outer source 
with side information coding and inner channel coding, both at rate R. If both coding are random coding 
that achieves the random coding error exponents for both source coding and channel coding respectively. 
The union bound of the error probability is as follows: 

Pr(a" / a"(b",y")) = Pr(c"^ ^ c"^(y")) + Pr(a" ^ 3(c"^(y"), fa"), c"^ = c"^(y")) (30) 

< Pr(c"^ ^ c"^(y")) + Pr(a" / 3(c"^(y"), b'')\d'^ = c"^(y")) (31) 

< 2-"(^-('f^'^^i-^)-^") + 2~"(''-^(^'-^-*«)"'") (32) 

where and converges to zero as n goes to infinity. (l30l) follows the union bound argument that a 
decoding error occurs if either the inner channel coding fails or the outer source coding fails. (|3TI) is true 
because conditional probability is large or equal to joint probability. Finally ( [32l ) is true because both 
the outer source coding and inner channel coding achieve the random coding error exponents. From (l32l) 
and that we can optimize the digital interface rate R between the channel coder and source coder, we 
know that the separate coding error exponent is 

max{mm{Er{R, Wy\x), eiiR, Pab}} = Eseparate{PAB, Wy\x) (33) 

This separate coding scheme is also discussed for joint source channel coding in [5]. A similar bound 

is drawn. We next show why the separate coding error exponent Eseparate{PAB,WY\x) is in general 

strictly smaller than the lower bound of E{Pab, in (ED- 

First, obviously, Eseparate{PAB, Wy\x) < max{min{£',.(i?, Wy\x)-, eu{R, Pab)}}- Secondly {Er{R, Wy 

R 

is monotonically decreasing, eu{R, Pab) is monotonically increasing, and both are continuous and convex 
as shown in Figure |4l This means that for rate R such that Er{R,WY\x) = eu{R, Pab)'- 

EseparateiPAB,WY\x) = EriR,WY\x) = eu{R,PAB) 

Now let R* be the rate to minimize {eu{R, Pab) + Er{R, Wy\x)}^ i-^- 

E{Pab, Wy\x) > eu{R*,PAB) + EriR*, Wy\x)- 
There are three scenarios. First if R* = R, then 

E{Pab,WyIx) > eu{R\PAB)+Er{R\WY\x) = 2Er{R,WY\x) = 2E,eparateiPAB,WYlx)- 

Secondly, if R* < R, 

EiPAB,WY\x) > EriR*,WY\x) > EriR,WY\x) = Eseparate{PAB ,Wy\x) ■ 

Finally if R* > R, 

E{Pab,Wy\x) > eu{R\PAB) > euiR,PAB) = Eseparate{PAB,WY\x)- 

So in all cases, the joint source channel coding error exponent E{Pab,Wy\x) is strictly larger than 
the separate coding error exponent EseparateiPAB, This is clearly illustrated in Figure ID 

Note: EseparateiPAB, Wy\x) '^^ achievable separate coding error exponent from the obvious separate 
coding scheme. What we prove is that this obvious one is strictly smaller than the joint source-channel 
coding error exponent. This is similar to the claim Csiszdr makes in [5]. It should be clear that the upper 
bound of any separate source channel coding error exponent is maxji{mm{Esp{R, Wy\x)j ^uiPj Pab}} 
which is comparable to (i33i . The proof hinges on the complete transparency between the source coding 
and channel coding, otherwise we have a joint coding schemes. A detailed discussion is in [14]. 
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VI. Conclusions 

We study the joint source channel coding with decoder side-information problem, with or without 
encoder side-information. This is an extension of Csiszar's joint source channel coding error exponent 
problem in [5]. To derive the lower bound, we use a novel joint source channel with decoder side- 
information decoding scheme. We further investigate the conditions under which the lower bounds and 
upper bounds match. A game theoretic approach is applied to show the equivalence of the lower and 
upper bound. This approach might be useful in simplifying other error exponents with a cascade of 
min-max operators , for example, the Wyner-Ziv coding error exponent recently studied in [10]. 

Appendix 

A. Proof of upper and lower bounds on Eboth{PAB^ Wy\x) 

We prove Proposition \T\ in this section. The upper bound and lower bounds are simple corollaries of 
the method of types and Theorem 5 in [5] respectively. 

1 ) Upper bound: Consider a distribution Qab, the joint source channel encoder observes the realization 
of the source (a", with type Qab, for the case where the decoder knows the side-information 6". There 
are|^ 2"'^^'^'5'*i^)~'^") many equally likely sequences G conditional on b^. These are the sequences with 
the same joint probability with 6" as the sequence a". Even knowing the joint type Qab (given by a 
genie) and the side-information 6", the decoder needs to guess the correct one from the channel output 
y". This is a channel coding problem with rate H{Qa\b) ~ ^n- 

Now consider the channel input x"(a", 6") where b"' is the side-information, notice that there are at most 
(n + l)!"^! many different input types, there is a type SxiQAB), such that more than (n + l)"!"^! = 2~"'^" 
fraction of the channel inputs given side-information 6" and the joint type of (a", 6") being Qab have 
type Sx{Qab)- For a channel Vy\x, such that the channel capacity of the channel given the input 
distribution Sx is smaller than H{Qa\b), i-C- 

I{Sx{QABy,VYlx) < H{Qa\b), 

then if the channel Wy\x behaves like Vy\x with the code book with type Sx{Qab), with high probably, 
the decoder cannot correctly decide from one of the 2"^(*3a|b) sequences. This is guaranteed by the 
Blowing up Lemma [6] or see a detailed proof in [7]. 

The probability that both the source behaves like Qa\b ^^d the channel behaves like Vy\x is 

2~n{D{QAB\\PAB)+D{VYix\\WYix\Sx{QAB))-el) (-34^ 

Notice that the source behavior Qab and the channel behavior Vy\x are arbitrary, as long as H{Qa\b) > 
*Here €'„ goes to zero as n goes to infinity, i — 1,2, 3. 
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I{Sx{Qab)', Vyix)' we can upper bound the error exponent as follows: 

Eboth{PAB,WY\z) 

< mill {D{Qab\\Pab)+D{Vy\x\\Wy\x\Sx{Qab))} (35) 

Qab,Vyiz:H{Qaib)>HSx{Qab);Vyix) ' ' 

= min{ min + ^(lV|xll^y|x|5x(QAij))} (36) 

= mini min {DiQAB\\PAB) + ^ mm I)(Fy|x||l^y|x|5x(QAB))}}37) 

R Qab:H{Qa\b)>R Vy\z:R>I{Sx{Qab)\Vy\x) 

< min{ min {D{Qab\\Pab) + Esp{R,WY\x)}} (38) 

R Qab-H{Qa\b)>R 

= imn{eu{R,PAB) + Esp{R,WY\x)} (39) 



(1351) is a direct consequence of (|34] ). In (I36b . we introduce the "digital interface" R, the equivalence 
in (l36l) and (|37]) should be obvious. (1381) and (l39l) are by definitions of the channel coding and source 
coding error exponents. □ 

2 ) Lower bound: Given a side-information sequence 6" which is known to both the encoder and the 
decoder. We partition the source sequence set based on their joint type with 6". The number of joint 
types rtin < {n + and denote by Q\q, i = 1,2, ...rUn the joint types. It should be clear that the 

Q\b^ ^^^^ ^^^^ same marginal distribution as 6". 

Let = {a" : (a", 6") € Q\b], i = l,2, ...m„. 

Obviously, ^j's form a partition of A"". And each set has size < 2^^^^^\^\ Now we can apply 

Theorem 5 of [5] as recited earlier: there exists a channel code /o,(/>o, such that for each a" G Ai{lP), 
i.e. gQ^^: 

p^ j^„(a") = ^Wy|x(y"|x''(a",6'^))l(a" / a"(6",y'^)) < 2~"(^'-(-f^(«^iB)'^^i^)-^"). (40) 
The joint source channel coding error probability is hence: 

a" ,6" y" 

< ^ ^ P4^(an^5"^2-"(-^'(^(Q-*i«)'^^i^)-^") (41) 

Qab {a" ,b")€QAB 
Qab 

Qab 

-n{mm{D{QAB\\PAB)+E^{H(QAiB),WYix)}-e'J 

< 2 '^■^^^^ (42) 

(|4T]) follows by substituting in ( |40b and the rest inequalities are by method of types, 0, so we can 
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lower bound the error exponent as 

Eboth{PAB,WY\z) > mm{D{QAB\\PAB)+Er{H{QA\B),WYlx)} (43) 

Qab 

= mm{ mill {D{Qab\\Pab) + Er{H{QA\B),WYix)}} (44) 

R Qab-H{Qa\b)=R 

= mm{ mill {D{Qab\\Pab)} + Er{R,WYix)} (45) 

R Q AB'-H{Q a\b) — R 

mill { min {D{Qab\\Pab) + Er{R,WY\x)}} (46) 

R>H{Pa\b) Qab:H{Qa\b)=R ' 

mill { min {D{Qab\\Pab) + Er{R,WYix)}] (47) 

R>H{Pa\b) Qab.H{Qa\b)>R ' 

mill {eu{R,PAB)+Er{R,WY\x)} (48) 

R>H{Pa\b) 

= imn{eu{R,PAB) + Er{R,WY\x)} (49) 

it 



(1431) is a direct consequence of (1421 ). in (1441) we again introduce the "digital interface" variable R. (1451) and 
(l48l) are by definitions of Er{R, Wy\x) ^^d eu{R, Pab) respectively. (l46l ) is true because Er{R, Wy\x) 
is monotonically increasing with R and for R < H{Pq\a)' 

min D{Qab\\Pab)>^= min I)(gAB||i^AB). 

Qab.H{Qa\b)=R Qab.H{Qa\b)=H{Pa\b) 

(l47l) is true because -D((5yiB||-PAB) is convex in Qab and the global minimum is Q*aq = Pab, but 
~ H{Pa\b) > which means the minimum point is on the boundary. Lastly (l49l) is because 
for R < H{Pa\b), ^u{R^ Pab) is constant at 0, while Er{R,WY\x) is monotonically increasing with 
R. □ 

B. Lower and upper bounds on E{Pab, Wy\x) 
We give the proof of Theorem \T\ here. 

1 ) Lower bound: From the definition of the error exponent, we need to find a encoding rule x : —>■ 
and decoding rule a : x y"- — > ^"such chat the error probability : 

Pr(a" a"(fa",y'^)) = ^ P^fiCa", 6") J] VFy|x(y"l^"(a"))l(«" y")) (50) 

is upper bounded by 2"^^"'^") where e„ — > 0, where E is the right hand side of (fT4l) . 

We first describe the encoder and decoder, then prove that this coding system achieves the lower bound. 

The encoder only observes the source sequence a". For all those sequences a" with type Qa, the 
channel input is x'^(a") that has type Sx{Qa), i-e. the channel input type only depends on the type of 
the source, where Sx{Qa) is the distribution to maximize the following exponent: 

min {D{Qab\\Pab) + D{Vy\x\\Wy\x\Sx{Qa)) + \I{Sx{Qa); Vy\x) - H{Qa\bT}. 

Qb\a,Vy\x 

The decoder observes both the side-information 6" and the channel output y", the decoder takes both 
the conditional entropy and mutual information across the channel into account: 

a"(6",y") = argmax/(a;"(a");y") -i?(a"|6") (51) 

a" 

We next need to show that there exists such a encoder/decoder pair that achieve the error exponent 
in (fT4l ). We also use the method of random selection of codebooks. We denote by C the set of the 
codebooks such that the codewords for a" G Qa all have composition Sx{Qa)- Obviously C is finite, 
we let C, be the random variable uniformly distributed on C. We use codebook c if C = c, i.e. we use 
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the codebooks with equal probability. The most important property of this codebook distribution is the 
point-wise independence of the codewords, for all a" G Qa and a" G Qa, for any two valid codewords 

e Sx{Qa) and G Sx{Qa) : 

^ ^ ' ^ ' ' ^ ^ ^ J K K J J \Sx{Qa)\\Sx{Qa)\ 
We calculate the average error probabihty on the whole codebook set C. Write the average error 
probabiUty as , then first we have: 

= E(Pr(a- ^ a"(6-,y-)) = ^^Pria^ ^T{b-,y^)), (53) 

C 

where i?(Pr(a" 7^ a"(6",y")) is the expected error probabihty over all codebooks under the codebook 
distribution (. 

For a fixed codebook c e C 

Pr(a" ^ a"(6",y")) 

a", 6" 

= E E (^'AB(a^6")Pr(a-^a"(6^y"))] 

= E E (PAB(a",6")5^W^y|x(y"|x"(a"))l'=(a"7^a"(6",y"))) 

= E E 

Qab {a" ,b'^)£QAB 

PAB{a^, 6") 5] T^y|x(y"k"K))l'^(a" 7^ 0^(6", y")) j (54) 

Vy\x y"':{x"'{a"),y"-)eSx{QA)xVY\x J 

For (a", 5") G Qab, so the source sequence a" has marginal distribution Qa, from the codebook 
generation we know that the codeword x"(a") G Sx{Qa)- For side-information 6" G we partition 
according to the joint type with 6": 

Q^^(6") = KG^":(a",6")GQ^^}. 

We partition Sx{Qa) according to the joint distribution with y". For a joint distribution Uxy s.t. 

Ux = Sx{Qa) and y" G ?7y: 

f/xy(QA,y") = {x" G Sx(Q^) : (x",y") G C/xy}. 

For (o",6") G (5a_b and channel output G 3^", s.t. (a;"(o"),y") G T^'IX' a decoding error is made 
if there exists a source sequence a" / a",s.t. a" G where Qab "^^y or may not be Qab and 

the code word x"(a") G C/xy(y", Q^)' where Ux = Sx{Qa) and y" G ?7y: 

ie. luxAX^Y) - H{Q^^j,) > I{Sx{Qa)\Vy\x) - H{Qa\b) (55) 
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Now we can expand the indicator function in (I54b as follows, for a codebook c: 



< min{l, l^(3a" / a" and a" G Q^^(6''), 

Qab, Ux^:Iu^^{X,Y)-H{Qa^^)>I{Sx{Qa);Vy\x)'H{Qa\b) 

s.t.x^{a^)eUxY{QA,y'')} (56) 

Under the uniform codebook distribution Q, for a" 7^ a", x"(a") is uniformly distributed in Sx{Qx) 
independent of x'^(a"), so for all and Uxy with the proper marginals(6" G Qb, ^^x = 'S'x(Q^) 
and G C/y) and satisfying (l55l) : 



£;(l(3a" / a" and a" G g^^(6"),s.t. x"(a") G C/xy (Q^, y"))) 
1^1 cec 

= Pr(3a" ^ a" and 5" G s.t. x'^(a'^) G Uxy{Qa, y")) 

< |QAB(&")|Pr( ^"(«") e t^xy(QA,y")|a" 7^ a" and a" G Q^b(6")) (57) 
- in- (un JUxY{QA,y'')\ 

2nH{Ux\Y) 

< 2"^"2"^(^-4i«) ^77^ (59) 

< 2-"(^('^^('3-^)'^^i^)"-^('3^i«)-'") (60) 

where e„ ^ 0. dSTl ) is by a union bound argument. dSSl ) is true because the codeword x"(d") is 
uniformly distributed in Sx{Qx)- < l59l) is by the method of types. (l60l ) is true because the condition 
in dSSl ) is satisfied. 

Combining (l56l ) and (l60b and noticing that the numbers of types of Uxy and are polynomials 
of n, hence sub-exponential, we have: 

1^1 cec 

< min{l, 2-"(^(^^(0-^)'^^i^)-^(Q-*i«)-^")} 

^ 2-"l-f('^^W-*);^^i^)--f^(Q-*is)-^"i+ (61) 

Finally, we substitute (|6T]) and (l54b into (1531 ). Notice that the number of types of Vy\x and Qab 
are polynomials in n and the usual method of types argument ( upper bounding the probabihty of 
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PAi?(a", b") G Qab) etc.), we have: 

IB iVy\X 

E 2 



< ^ 9-n(D(QAi3||P^B)+£'(VV|x||VFi.|x|S'x(QA))+|/(5x(QA);VV|x)-H(QA|B)| + -ei-e^) 



AB ,VY\X 



< 2 



-n(minQ^^,^, ,^{D(Q^B||PAB)+I5(VV|x||Vyvix|Sx(QA))+|/(Sx(QA);VV|x)-J^(QA|B)| + }-4) 



where ^ for i = 1,2,3. Notice that p" is the average error probability of the codebook set C, so 
there exists at least a codebook c, such that the error probability is no bigger than . 
Now we lower bound the achievable error exponent by 



min {D{Qab\\Pab) + D{Vy\x\\Wy\x\Sx{Qa)) + Vy\x) - H{Qaib)\^} 

Qab,Vyix 

= min min {D{QAB\\PAB)+D{VYix\\WYix\Sx{QA)) + \I{Sx{QAy,VYix)-H{QA\B)\^} 

Qa Q b\a,Vy\x 

= min max miri {I)(g^B|lPAB) + Z?(^y|xl|H^y|x|5x(QA)) + |/(5x(Qa); VV|x) " i^(QA|B)l+} 

Qa Sx(Qa) Qbia,Vyix 

The last equality is true because that the codeword composition Sx{Qa) can be picked according to 
the source composition Qa- And by our code book selection we always pick the composition to maximize 
the error exponent 

min {D{Qab\\Pab) + I)(^y|xl|VFy|x|5x(QA)) + IHSxiQA); ^y|x) " ^(Qaib)^}- 

Q b\a,Vy\x 

Here we sUghtly abuse the notations where Sx{Qa) is always the optimal distribution to maximize 
the above exponent given Qa- 

The lower bound on E{Pab, Wy\x) in Theorem ([T]) is just proved. ■ 

2) Upper bound: |Z| First we fix the source composition Qa, there are 2"(^('5-*)~'^") sequences in A"' 
with type Qa- When the encoder observes the source sequence a", it has to send a code word x"(a") 
to the channel Wy\x- There are at most (n + 1)1^1 different types, so at least 

2n{H{QA)-ei) 

(n + 1)1^1 

of the codewords for a" G Qa have the same composition, we write this composition Sx{!Qa), and 

A^ = {a" G Qa : x"(a") G Sx{Qa)}, where \Ai\ = 2"(^W^)-^"). 

Now we fix the conditional type Qb\a^ so we have the marginal Qb and the joint distribution Qab 
determined by Qa and Qb\a- Write 

QA\Bm = {a" : (a", 6") G Qab} and Qb|a(«") = {b" : (a",6") G Qab}- 

Obviously \Qb\ = 2"(^(Q«)-^» and for all = 2"(-f^(Q^i«)-<), for all a": |gB|A(a")| = 

2n(H(QB|A)-e^')_ 

'in this proof, ej^ > and ej, ^ 0, i = 1, 2, 3, 4, 4', 5, 6 and 7. 
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Let Bi = {6" G Qb : i<3A|i?(fe") fl ^il > 2"(^(Q^i«)"^")}, where = + e^' + i We show next 
that the size of Bi is of the order 2''^(Q«). 

Let ABi = {(a", 6") : a" G Ai and (a", 6") G Qab}, we compute the size of ABi from two different 



ways. 
First 



Secondly 



\AB,\ = \A,\\QBiAian\ = 2"(^W-)-^"-^"). (62) 

\ABi\ = \{{a'', V') : 6" € Si, a" G Ai and (a", 6") G Qab} 

|J{(«", 6") : fc" G - 5i, a" G Ai and (a", 6") G g^i?}! (63) 

< |i?i||QA|B(m + IQi?-5i|2"(^^^^'^^-^") (64) 

< |iJ-|^|2"-'^('3-*i^) + 2""'^^*^^^2"^"'^*^*^'^i^^~'^"^ (65) 



(1631) is by the definition of ABi and i^i, (1641 ) is by the definition of Bi, (1651) is true because all e^'s are 
positive. 

Combining (l62l) and (l65l) and use the fact that = + + ^, we have: 

]^q^]^2'^h(Qa\b) > 2"(^(Q-4b)-4-<') _ 2"^('3s)2"(-^('5^i^)~^"^ 

^ 2n{H{QAB)-el-4) X 1 

2' 

Hence > 2"(^(^-)-^"-^" = 2"(^(Q«)-4). 

Now we consider the decoding error of the following events and show that this error events gives us 
an upper bound on the error exponent stated in this theorem: 

Source and side information pair AB* = {(a", 6") : a" G Ai, 6" G 5i, (a", 6") G Qab]- 

First, for each (a", 6") G yl5*: 

PAB{a^,b") = 2^"(-^('3-*«ll-^^«)+-'^('3^«)). 

Secondly, the size of AB* is lower bounded as follows from the definition of Bi and the lower bound 

on \Bi\: 

\AB*\ > X 2"(^(^^i«)-"")} 

So obviously the probability of AB* is 

Pab{AB*) = |^_g*|2~"(-°W-*Bll-f'-'i«)+-'^W-'i«)) > 2~"(-^(Q^sll^'^s)+24), (57) 

Thirdly, if the side-information is 6" G -Bi there are at least 2"(^('3A|f3)-e^) jjj^jjy ^^'^ ^^^j^ jj^^j 
(a", 6") G Qab, that is, there are at least 2"*^^*^'5"*i^)~^") many source sequences with the same likelihood 
given the side-information 6" (even there exists a "genie" that tells the decoder that the joint distribution 
of (a", 6") is Qab)- Furthermore, the channel input codeword x"(a") for these source sequences all 
have composition Sx{Qa)- Hence we have a channel coding problem with rate H{Qj^\b) ~ ^^'^ fixed 
input composition Sx{Qa)- This is the standard channel coding sphere packing bound studied in [7]. 

So if 6" G Bi, then average error probabiUty for (a", 6") G AB* is at least: 



18 



-n{ min {D(VY\x\\WY\x\Sx(QA))}+el) 

2 Vy^x-'<.Sx«3a):Vy\x)<"'.Qa\b)-'1. 

-n{ min {D{VY\x\\WY\x\Sx{QA))}+el) 

> 2 '^Y\X--nSx{QAy,VY\x)<HiQA\B) ("gg^ 

where and goes to zero as n goes to infinity, hence — > because /(^^(Qa); ^y|x) is continuous 
in Vy\x ^rid -^(^y|xll^y|xl'S'_x(<5A)) is convex in 

Finally we combine (|67] ) and (|68] ). and notice that the above analysis is true for any(adversary) distri- 
bution of the source Qa, and any(optimal) channel codebook composition Sx{Qa), and any(adversary) 
Qb\a after Qa and Sx{Qa) are chosen, the error probability is lower bounded by: 

-n(mm max rain {D(Q ab\\Pab)+ min {D{VY\x\\WY\x\Sx{QA))}}+2el+el) 

2 Qa Sx{Qa)Qb\A VY\x-'(Sx{QA),VY\x)<HiQA\B) 

-n(min max min {D(QAB\\PAB)+D{VY\x\\WY\x\Sx(QA))}+2el+tl) 

_ 2 'Sa Sx(QA)QB\A''^Y\X--'iSx(QA)<VY\x)<H(QA\B'> 

Both e,^ and converges to zero as n goes to infinity, the upper bound in Theorem [T] is just proved. ■ 

C. Proof of Corollary |7] 

The proofs for both lower bounds and uppers with the "digital interface" are similar. 

1 ) Proof of f lTTl ). the lower bound: By introducing the auxiliary variable R to separate the source 
coding and channel coding error exponents and the definition of error exponents, the following equalities 
should be obvious. 



E{Pab,Wy\x) 

> min max ^ min D{Qab\\Pab) + D{Vyix\\Wyix\Sx{Qa)) + IHSxiQA); Vy\x) - H{Qa\b)\^ 

Qa 5x(Qa) Qb|a,VV|x 

= min max min 

Qa Sx(Qa) R 

{ min D{QAB\\PAB) + D{VYix\\WY\x\Sx{QA)) + \I{Sx{QAy,VY^x)-R\^} 

Qb\a,Vy\x-H(Qa\b)—R 

= min max min{ min D{Qab\\Pab) + Er{R, Sx{Qa),Wy\x)} (69) 

Qa Sx{Qa) R Qb\a.H{Qa\b)=R 

= min ^s.^ mui{e'u{R,PAB,QA) + Er{R,Sx{QA).WY\x)} (70) 

Qa Sx(Qa) R 

where Er{R, Sx{Qa), Wy\x) is the standard random coding error exponent for channel Wy\x at rate R 
and input distribution Sx{Qa), while e'jj{R, Pab, Qa) is a peculiar source coding with side-information 
error exponent for source Pab at rate R, where the empirical source distribution is fixed at Qa- That is 
for Qa 

e'u{R,PAB,QA) = min D{Qab\\Pab) 

Qb\a:H{Qa\b)=R 

dTOl ) needs more examination. It is obvious that 

e'u{R,PAB,QA) > min D{Qab\\Pab) = euiR,PAB,QA)- 

Qb\a'-H\Qa\b)^R 

where eu{R,PAB,QA) is defined in (fTSl ). Now (ITOl ) becomes 

E{Pab, Wy\x) > min max mm{eu{R, Pab, Qa) + Er{R, Sx{Qa), Wy\x)}- (71) 

Qa SxiQA) R 
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2) Proof of fl76l). the upper bound: Similar to the proof for ([TT]) . we have the following equalities: 
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where Esp{R, Sx{Qa), Wy\x) is the standard sphere packing bound defined in dSjl and eu{R, Pab, Qa) 
is defined in ([HJl. □ 
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